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CLAIMS 
[Claim(s)] 

[Claim 1] The infrared camera which is a MAMMA scene interface system using the image of a 
paper fingertip, and obtains the image on a desk, The image by said infrared camera is inputted 
and it has the processing system to process. Said processing system The MAMMA scene 
interface system characterized by searching for the coordinate on the desk of the fingertip 
which specified with pattern matching with the pattern which prepared the fingertip beforehand 
from the hand field which extracted the hand field using temperature and was extracted, and 
was specified from the image. 

[Claim 2] It is the MAMMA scene interface system characterized by said pattern prepared 
beforehand being a circular pattern in a MAMMA scene interface system according to claim 1. 
[Claim 3] It is the MAMMA scene interface system characterized by including the processing 
recognized to be pointing actuation when the number of the fingertips which asked for said 
processing system further in the MAMMA scene interface system according to claim 1 or 2 is 
one. 

[Claim 4] It is the MAMMA scene interface system characterized by performing control which 
doubles the camera station of said camera with the coordinate on the desk of the fingertip when 
it has been recognized as said processing system being said pointing actuation, while having 
further the camera which can perform control of a camera station in a MAMMA scene interface 
system according to claim 3. 

[Claim 5] It is the MAMMA scene interface system which said processing system captures the 
image photoed with said camera in said MAMMA scene interface system according to claim 4, 
and is characterized by extracting a bar code and performing bar code recognition processing 
from the captured image. 

[Claim 6] It is the MAMMA scene interface system characterized by for said processing system 
inputting voice from said voice input means while having a voice input means further in said 
MAMMA scene interface system according to claim 1 to 5, and performing speech recognition 
processing. 

[Claim 7] The MAMMA scene interface system which is equipped with a projector and is further 
characterized by projecting a computer screen on a desk by said projector in a MAMMA scene 
interface system according to claim 1 to 6. 

[Claim 8] The record medium which stored the program to which the processing performed with 
a MAMMA scene interface system according to claim 1 to 7 is made to carry out to computer 
system. 

[Translation done.] 
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DETAILED DESCRIPTION 


[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] About a MAMMA scene interface, especially, this invention does not 
make use of a computer conscious of the existing actuation, but thinks a users amenity as 
important, and relates to the interface to which a users degree of freedom is not reduced by 
attaching a measuring instrument etc. in a user's body. 
[0002] 

[Background of the Invention] A computer permeates our life deeply and the use gestalt is also 
various. Before, a very small computer which was not considered is carried or use in a location 
which is embedded in a surrounding environment, glances and does not turn out to be it is also 
performed. It is thought that it meets and develops into two flow, the gestalt attached in a man 
like a wear rubble personal computer or PDA in the future and the gestalt united with the 
environment around [, such as a desk, and a wall, a room, ] a user. When the MAMMA scene 
interface was considered in such flow, the interface represented by the present GUI etc. had to 
bind the user to the interface, and the interface also had to be taken care of apart from the 
activity to carry out originally. So, in the next-generation interface, the interface which can 
perform more direct and more natural actuation to the activity which a user wants to do is 
searched for. 

[0003] Then, in the informational retrieval and the use using a computer, for mitigation of the 
burden by a user s interface, a computer supports an activity and actuation of a user in the real 
world, and it has a hope for real world orientation and the interface by situation recognition as 
an interface of the next generation which can be used more comfortably than the interface of 
the conventional type only by the keyboard or the mouse. As for a real world orientation user 
interface, a computer always recognizes the situation of the user in the real world, an intention 
of a user is drawn up, and a computer supports an activity. This aims at implementation of the 
transparent interface in which actuation of the existing computer is not impressed. This real 
world orientation interface is applied to a paper activity, and development of the real world 
orientation interface (Enhanced Desk) integrated by the "desk" is furthered as an example of a 
****** world orientation interface. As an example which took notice of fusion of a paper 
activity and an activity by computer, Digital Desk (for example, pool "Bit separate volume visual 
aiming at interface-post GUI refer to KYORITSU SHUPPAN, Chapter 2.1, and pp.24-44 
grade) is known well. At Digital Desk, the computer screen projected on the desk is operated by 
a fingertip etc. A user can cut off and copy the illustration projected on the desk, or can 
calculate with the calculator projected on the desk. 

[0004] Paying attention to the paper document on a desk, there is also an example which has 
tried integrated use of a paper document and electronic intelligence (M. Kobayashi and H.Koike: 
Enhanced Desk, Integrating Paper Documents and Digital Documents; Proceedings of 1998 Asia 
Pacific Computer Human Interaction, and pp.1 67-1 74 reference (1998)). Although the bar code 
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beforehand given to the paper document for matching of a paper document and electronic 
intelligence was used in this example, since it was required to observe a bar code in the above 
magnitude to some extent the large-sized bar code had to be used. Moreover, in order to 
recognize a user's hand field, the beige extract was performed, but when it will have been 
incorrect-recognized if the near body was beige on the desk, or the image was projected on a 
users hand, there was a trouble that a hand field extract did not work. Always assuming the 
sense of a specific hand also about recognition of a user's fingertip location, the fingertip was 
the problem with much [ and ] constraint of considering as one point that the insufficiency of 
recognition precision was big. And since software was performing a series of processings, 
real-time processing was impossible. 
[0005] 

[Problem(s) to be Solved by the Invention] The purpose of this invention is information interface 
system implementation integrated by the "desk" as an interface united with the environment 
around a user. In office, the activity using PC is done on a desk, and the activity using a paper 
document is done on coincidence in many cases. The information interface system distribution 
which can treat the activity on these desks integrative by making a computer observe a desk 
top is performed. 
[0006] 

[Means for Solving the Problem] The infrared camera which this invention is a MAMMA scene 
interface system which uses the image of a paper fingertip, and obtains the image on a desk in 
order to attain the above-mentioned purpose, The image by said infrared camera is inputted and 
it has the processing system to process. Said processing system It is the MAMMA scene 
interface system characterized by searching for the coordinate on the desk of the fingertip 
which specified with pattern matching with the pattern which prepared the fingertip beforehand, 
and was specified from the hand field which extracted and extracted the hand field from the 
image using temperature. Said pattern prepared beforehand can also be used as a circular 
pattern. Thereby, the recognition precision of a fingertip can be raised and, moreover, 
recognition processing can be performed at a high speed. 

[0007] Furthermore, when the number of said fingertips for which it asked is one, it can be 
recognized as it being pointing actuation, and it is this pointing actuation and pointing actuation 
performed with a mouse etc. can be performed. Furthermore, while having the camera which can 
perform control of a camera station, control which doubles the camera station of said camera 
with the coordinate on the desk of the fingertip can be performed by performing said pointing 
actuation. Thereby, the image photoed with said camera is captured, from the captured image, a 
bar code can be extracted and bar code recognition processing can also be performed. 
Furthermore, while having a voice input means, a command with voice etc. can be inputted by 
inputting the voice by said voice input means, and performing speech recognition processing. 
Furthermore, said projector can also perform interactive processing with a computer screen by 
projecting a computer screen on a desk by having a projector. The record medium which stored 
the program to which above-mentioned processing is made to carry out is also this invention. 
[0008] 

[Embodiment of the Invention] The MAMMA scene interface system of this invention has the 
following descriptions. 

(1) Extract a users hand field to stability by measuring the synchrotron orbital radiation from a 
user's skin field using an infrared camera. 

(2) The use of template matching based on normalization correlation etc. realizes a high-speed 
trace of a user's fingertip. 

(3) In order to recognize a paper small bar code, carry out the expansion trace of a user's 
fingertip circumference using a pan tilt camera with a zoom function. 

(4) Perform use of image-processing hardware, and application of distributed processing for 
improvement in the speed of processing. 
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Below, the operation gestalt of this invention is explained with reference to a drawing focusing 
on these descriptions at a detail. 

[0009] Drawing 1 is the external view of the en hunger strike disk (Enhanced Desk) which is the 
operation gestalt of this invention. As shown in drawing 1 , a projector 30 is attached above the 
usual desk 10, the screen of a computer (not shown) is reflected in a mirror 44, and Screen 12 
is projected on a desk. Two cameras which observe a desk top were installed. One set is an 
infrared camera 50, turns and installs a top from desk 10 left-hand side, and performs paper 
photography throughout a period of surface reflecting mirror 42. One more pan tilt camera 20 
with a zoom function can be installed in the upper right direction of a desk, and can expand and 
photo a paper part. Moreover, although not illustrated, the loudspeaker and the microphone are 
also installed and it also has the function of I/O, such as voice. 

[0010] Drawing 2 shows the system configuration Fig. in an operation gestalt In drawing 2 , the 
projection image generated with computer system 60 is projected by the projector 30 on a desk 
10 through the surface reflecting mirror 44. Moreover, the posture of the pan tilt camera 20 
which is picturizing the image on a desk 10 etc. is controlled by computer system 60. The image 
picturized with the pan tilt camera 20 is also captured and processed in computer system 60. 
The image on the desk picturized with the infrared camera 50 is also captured and processed by 
computer system 60. The flow of the main processings with the en hunger strike disk of this 
invention is shown below. 

** Extract a user's hand field from the image of an infrared camera 50. 
** Detect a user's fingertip point from the extracted hand field. 

** Change into a paper position coordinate from the position coordinate of the fingertip point on 
a camera image using the projective-transformation parameter for which it asked beforehand by 
the calibration. 

** Take the object and information which were projected on the desk based on paper fingertip 

positional information, and interaction. 

These processings are explained to a detail below. 

[0011] the extract of a <extract of hand field> hand field — a background — generally the beige 
extract with difference or a color camera is used. However, it is difficult to extract a hand field 
by such technique depending on the condition of a background in many cases. In the work 
environment assumed especially by this system, in order it is possible to open a paper 
document, books, etc. and to project electronic intelligence by the projector on a desk, there is 
a trouble that the color of a users hand field is not fixed etc. So, in this system, in order to 
extract a hand field to stability, the infrared camera 50 was used. From the image of the infrared 
camera which is observing the desk top, a hand field can be started based on people's 
temperature. The situation of a paper hand can be obtained as an image of 256 gradation of 
NTSC using an infrared camera by taking a photograph near people's temperature (30 degrees C 
- 34 degrees C). This image is binary — ization-processed with a fixed threshold, and a hand field 
is extracted. Extract processing of a hand field is explained to drawing 3 based on the image 
from an infrared camera. Drawing 3 (a) is an image from an infrared camera 50. The resolution at 
the time of incorporating with computer system 60 is 256x220 pixels. And the image near 
people's temperature (30 degrees C - 34 degrees C) is extracted from this image. Drawing 3 (b) 
showed this, and it extracts and makes the part of a hand field binary. As shown in drawing 3 
(b), it is not influenced by change of a background or lighting by this technique, but only a hand 
field can be extracted to stability. For example, NIKON thermal BISHON (LAIRD 3 A) can be used 
for an infrared camera. This camera can photo the range to -20 degrees C - 50 degrees C in 
field time 1 / [ 768x465 pixels of effective pixel numbers, and ] 60 seconds. 
[0012] detection of a Recognition of fingertip location> fingertip location has a user's hand on a 
desk, and the distance of the hand of a camera and a user can detect a fingertip location by the 
template of fixed magnitude from seeing by about 1 law and the upper magnitude not changing 
extremely. Generally recognition of a fingertip location is detected using a circular template 
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based on the profile of a fingertip configuration being close to a circle by performing pattern 
matching based on normalization correlation around a hand field. A user s hand is on a desk, 
from the magnitude on appearance not changing extremely, the object of pattern-matching 
processing is limited only within the limits of [ fixed ] a hand field, and improvement in the speed 
of processing is attained. Recognition of this fingertip location is explained using drawing 4 . 
Pattern-matching processing performs coincidence detection using a correlation value using a 
template with a magnitude of 15x15 pixels shown in drawing 4 (a) to the 256x220-pixel image 
(refer to drawing 3 (b)) which started and made for example, the hand field binary. When many 
points that a correlation value is high are found after pattern-matching processing in the 
location which approached too much, it leaves the point of them that a correlation value is the 
highest, and the remainder is excepted from a fingertip candidate point. Since some fingertip 
candidate points are acquired by this, the point of having been incorrect-recognized other than 
a fingertip point is excepted. As shown in drawing 4 (b), this processing investigates the pixel 
corresponding to eight perimeters (four rectangular top-most vertices and point of each side 
dividing into two) of a template, and judges whether it is a fingertip point When the fingertip has 
run through the template by this processing or anything does not have a pixel appropriate for a 
finger in (ERROR of drawing 4 (b)), and the perimeter of a template, it excepts from a fingertip 
point candidate. Finally, as shown in drawing 4 (c), a fingertip candidate point is adopted as 
order with a high correlation value to five, and it considers as a user's fingertip point. It is 
carrying out using the Hitachi image-processing board IP 5010 as an example of the fingertip 
detection processing in this system. IP5010 is the image-processing board which had the 
memory for 40 screens by monochrome shade image, and is the board which can perform 
binary-izing and an image processing called difference at a high speed by hardware between the 
images saved on memory. OS supports WindowsNT (trademark) and Linux and many 
image-processing functions are prepared as a library. In this example, Linux was used for OS in 
view of compatibility with a network. 

[0013] In order to know whether the location detected in the <amendment of camera image 
coordinate and desk Kamitaira side coordinate> infrared image is equivalent to which location on 
a desk, amendment of an infrared camera image coordinate and a desk Kamitaira side 
coordinate is performed. Here, some corresponding points of an infrared camera image 
coordinate and a desk Kamitaira side coordinate are defined, and correspondence between an 
infrared camera image coordinate (x y) and a desk Kamitaira side coordinate (x\ yO is used and 
expressed for projective transformation like a formula (1) based on the relation. 
[Equation 1] 

' x l = c x -x + c 2 y + c % xy + c A 

0) 

y' = c 5 x + c 6 y + c 7 xy + c s 

In order to ask for corresponding points, it explains using drawing 5 explaining measurement of 
infrared camera corresponding points. The system of coordinates of a desk Kamitaira side are 
set up to a desk 10. For this reason, the calibration plate 70 is created in the location of a paper 
target point, and a desk upper seat label system is set up by placing this on a desk. In the 
infrared camera which starts a hand field, since a temperature gradient is outputted as an 
image, as shown in drawing 5 , a miniature lamp 72 is embedded at the corresponding points of 
this calibration plate, the turned-on electric bulb is photoed with an infrared camera, and 
corresponding points are measured. A projective-transformation parameter (c1-c8) is 
determined based on the relation after measuring the group of corresponding points. The 
distortion of an image, a gap of a location, etc. can be amended by changing using this 
parameter. In order to obtain this projective-transformation parameter, there should just be at 
least 4 sets of corresponding points, but in this system, in order to ask stability for a solution, 
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as shown in drawing 5 , 9 sets of corresponding points (miniature lamp) were prepared. It asked 
for the parameter of conversion from these 9 sets by solving the above-mentioned 
simultaneous equations using a singular-value-decomposition method. In this system, paper 
system of coordinates and the system of coordinates of the image projected on the desk by the 
projector are made the same for processing mitigation, and the position coordinate of the value 
of a desk Kamitaira side coordinate and the pixel of a projector projection image is made equal. 
Thereby, in order to project the image of a projector on the fingertip location on a desk 
Kamitaira side, mitigation of processing can be aimed at that what is necessary is just to only 
project the image which drew the object on the coordinate value on a desk Kamitaira side 
coordinate. 

[0014] In a <bar code recognition by gaze of the fingertip circumference> book system, when 
the number of the fingertips of the user who has recognized by performing an image processing 
is one, a user can also judge with carrying out pointing actuation. It can use instead of pointing, 
such as a mouse, by recognizing it as it being pointing actuation by pointing out only by the 
index finger, and acquiring the coordinate of this fingertip by this. It can also be considered that 
it is the field which the circumference of the fingertip which is carrying out this pointing 
actuation is observing. The pan tilt camera 20 which is this application explains below the 
processing which pursues the attention field of one fingertip. 

[0015] (Flow of fingertip trace processing) The flow of processing of the fingertip trace in the 
pan tilt camera 20 is shown below. 

** Measure corresponding points among 2 flat surfaces of a desk Kamitaira side and a pan tilt 

drive flat surface, and compute the projective-transformation parameter. 

** Measure the location of the fingertip point on an infrared camera image by the image 

processing. 

** Change the location of the fingertip point on an infrared camera image into the coordinate on 
a desk Kamitaira side by projective transformation. 

** Use projective transformation and change the location of a paper fingertip point into a pan 
tilt drive flat surface. 

** Send out the coordinate on a pantilt drive flat surface to a camera in VISCA code, and make 
it turn [ camera ] to a user's fingertip point 

** Process the image of the camera suitable for a fingertip point 

[0016] (Calculation of a projective-transformation parameter) In this system, it should just be 
turned out whether a paper fingertip location is located on the pan tilt drive flat surface of a 
camera 20 in which coordinate location. Since it can know which coordinate value the pan tilt 
camera 20 has taken on the drive flat surface now, this processing is performed using this. 
Although it explained having performed amendment of a camera image coordinate and a desk 
upper seat label previously, also in the fingertip trace, coordinate amendment with a camera 20 
is performed using the same technique. Therefore, first, since the desk upper seat label of a 
fingertip is known by old processing, the location on the drive flat surface of the pan tilt camera 
20 corresponding to the desk top location is computed. Four or more sets of corresponding 
points are measured. This measurement adjusts a location so that a pan tilt camera may catch 
paper corresponding points at the core, and it measures the coordinate value on the pan tilt 
drive flat surface in that time as corresponding points. Previous projective transformation is 
used based on these corresponding points, and it asks for the parameter changed into the 
coordinate on a pan tilt drive flat surface from a desk upper seat label. The camera 20 for a 
fingertip trace has the function in which a camera can be turned in the direction by specifying a 
coordinate. A user s fingertip is made to pursue using this function. As an example, the tracking 
camera 20 for a fingertip trace is perpendicularly. - They are 7794-7794, and a horizontal 
direction. - Pan tilt control of a camera is performed by using a Sony EVI-G20 pan tilt camera 
with the pan tilt drive flat surface of the magnitude of 15570-15570, and transmitting the VISCA 
command to a camera from the serial port of PC. 
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[0017] (Fingertip trace processing) Now, fingertip trace processing is explained. By the 
above-mentioned image processing, the location of the fingertip point on a desk 10 is measured 
for the image from an infrared camera 50. While changing the location of the fingertip point on 
an infrared camera image into the coordinate on a desk Kamitaira side by projective 
transformation, it recognizes that it is only one fingertip. With an above-mentioned parameter, 
projective transformation is used and the location of the fingertip point on the desk for which it 
asked is changed into the pan tilt drive flat surface of a camera 20. The coordinate on a pan tilt 
drive flat surface is sent out to a camera in VISCA code, and it is made to turn [ camera / 20 ] 
to a user s fingertip point Thereby, the image of the camera 20 which turned to the fingertip 
point can be captured. 

[0018] (Recognition of a bar code) The image of this camera 20 is used for below, an expansion 
trace of a fingertip is performed to it, and the example which recognizes small bodies, such as a 
bar code, is explained to it using drawing 6 . Drawing 6 performs an expansion trace of a 
fingertip and shows signs that bar code recognition is performed. On actual bodies (a paper 
document, books, etc.), it can recognize by attaching a bar code, and a link with electronic 
intelligence can be created on them. The 2-dimensional matrix code was used for the bar code. 
This two dimensions bar code can calculate an objective posture from a code, if the location not 
only with the class of code but a code and the location where the objective code is stuck since 
the sense etc. can be obtained from a code are memorized The example of a two-dimensional 
bar code is shown in drawing 6 (a). Such a bar code will be recognized if it exists in the image 
photoed with the pan tilt camera 20. In order to recognize, the magnitude of extent which is 
among a camera image is required, but since the image which the pan tilt camera 20 expanded is 
photoed, it is possible to recognize to a thing with a magnitude of about 1.5x1 .5cm also at the 
bar code stuck on the document set on the desk. On a desk, by pointing actuation of a user, if a 
bar code is pointed at, the code can be recognized and the interaction corresponding to it can 
be raised. Signs to this bar code that the expansion trace of a fingertip was performed are 
shown in drawing 6 (b). Drawing 6 (b) shows the image which made the image of a camera 20 
binary. First, the location of the fingertip point on the image from an infrared camera 50 is 
measured by the image processing. Thereby, the position coordinate of having pointed out with 
one finger and its fingertip point can be acquired. Next, the location of the fingertip point on an 
infrared camera image is changed into the coordinate on a desk Kamitaira side by projective 
transformation. Projective transformation is used for a pan tilt drive flat surface, the location of 
the fingertip point on the obtained desk is changed, the coordinate on the acquired pan tilt drive 
flat surface is sent out to a camera 20 in VISCA code, and it is made to turn [ camera / 20 ] to 
a users fingertip point. And the image ( drawing 6 (b)) of the camera 20 which turned to the 
fingertip point is processed, a bar code image is recognized, and reading processing of a bar 
code is performed. Recognition processing of a bar code can incorporate for example, a video 
image in the magnitude of 320x240 pixels as an example from the video input terminal of 02 
which is the system of SGI, can process binary-ization etc., and can be recognized using a bar 
code recognition library. The image input is using the video library of SGI. Moreover, software is 
performing bar code recognition processing and it can perform at the rate of 10 - 15 f/s extent. 

[0019] Speech recognition is used for a <speech recognition> book system as an auxiliary role 
of interaction with a user. Here, it uses only in order to make the keyword for changing the 
operation modes, such as pointing actuation of a user, migration of an object, and rotation, 
recognize. Speech recognition is performed using the speech recognition engine of IBM 
ViaVoice. In this system, when a user utters the word registered beforehand, in order to raise a 
certain interaction, the speech recognition process has always sent the class of recognized 
word as a message to the message server. As for the message sent to the server, it is possible 
for an information presentation process to raise the interaction corresponding to reception and 
its word on a desk. 
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[0020] In the <distributed-processing> book system, it roughly divides and four processings of 
fingertip detection and the fingertip trace processing with a pan tilt camera, two-dimensional 
bar code recognition processing, speech recognition processing, and information presentation 
processing are performed. In this system, the machine of the plurality for improvement in the 
speed is performing distributed processing for each processing. The schematic diagram of 
distributed processing is shown in drawing 7 . Fingertip detection and the fingertip trace 
processing with the pan tilt camera 20 are for example, Pentium(trademark) II. It is carrying out 
using the machine which introduced Linux and the image-processing board IP 5010 into the 
personal computer 64 of 450Mhz. According to this process, a user's paper number of fingertips 
and paper location were measured, and the desk upper seat label location of a fingertip is sent 
out to the message server 68. Moreover, the camera 20 is controlled to become a fingertip 
location on a user s desk about the direction of a look of the pan tilt camera 20. By 
two-dimensional bar code recognition processing, it is SGI about the image of the pan tilt 
camera 20. It incorporates to the video input of 02 system 66, and the two-dimensional bar 
code is recognized by software processing. A recognition result is always sent out to the 
message server 68. Speech recognition is processing by introducing WindowsNT and ViaVoice 
into the personal computer 62 of Pentiumll 330Mhz. When the word which is made to recognize 
all the voice about which the user spoke toward the microphone, among those has been 
registered beforehand has been recognized, the specific message is sent out to the message 
server 68. Finally the information always stored in the message server 68 is taken out, and there 
is an information-display process which creates the interaction and image corresponding to it 
So to speak, this is an application process, and is SGI. It is processing by 02 system 66. In the 
message server 68 which performs the communication link during each processing, it is tuple 
space communication system TS. System/s and a common name Linda are used. This system 
68 has realized the communication link by sharing the space called a tuple tooth space on a 
network, and exchanging the message called the tuple which considered the character string of 
arbitration as the set through that space. Since it is asynchronous communication, each 
process can operate independently completely and there is an advantage that an addition and 
modification of a new process are easy. In this system, at the practical rate of per second 20 or 
more frames, it was stabilized and the fingertip was able to be detected. 
[0021] A user can recognize the alphabetic character written with the finger using pursuing 
pointing actuation of the user on a <example of application> (simple character recognition) desk. 
For example, if a figure is written in from a projector 30 with a finger to the grid projected on 
the desk, the figure can be recognized and the figure which wrote can be displayed by the 
projector 30 on a desk. It can use in order to give semantics other than simple pointing to a 
users fingertip. 

(Other application) As other application, not only the positional information of a user's fingertip 
but the gesture recognition using the locus of five fingertips etc. is possible. Since the 
photomacrography with a camera became possible and it is thought that the small alphabetic 
character of space can also be recognized, character recognition is used, and although the 
function for integrative use of a paper document and electronic intelligence is added, it can do. 
Moreover, construction of the communication environment through the desk Kamitaira side by 
sharing the two desk top of a remote place for this system as application to a remote 
coordination activity can also be performed. This invention may be applied to the client/server 
system which consists of not only an above-mentioned distributed computer system but two or 
more systems, or a stand-alone system. The configuration of this invention is realizable by 
reading and performing a program by the system from the storage which stored the program 
about this invention. There are a floppy (trademark) disk, CD-ROM, a magnetic tape, a ROM 
cassette, etc. in this record medium. 
[0022] 

[Effect of the Invention] As mentioned above, if this invention is used, an interface required for 
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the desk ''Enhanced Desk" implementation which can treat the electronic intelligence and the 
actual body on a desk integrative is realizable. Since the expansion trace of the fingertip was 
carried out with the camera, recognition of a small bar code was attained and it became 
unnecessary moreover, to attach a big bar code to the body on a desk unnaturally. As for a 
user, it is more nearly intuitive than a conventional mouse and a conventional keyboard to move, 
rotating 3D object projected for example, on the desk by using the interface of this invention by 
the fingertip, or to display the homepage related by pointing at the bar code attached to books 
etc., and he can perform reality and the interaction linked closely between electronic 
intelligence. 

[Translation done.] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the 
original precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 


DESCRIPTION OF DRAWINGS 


[Brief Description of the Drawings] 

[Drawing 11 It is the external view of an en hunger strike disk. 

[Drawing 2] It is the system configuration Fig. of this invention. 

[Drawing 3] It is drawing showing the extract of a hand field. 

[Drawing 41 It is drawing explaining fingertip recognition. 

[Drawing 51 It is drawing explaining measurement of corresponding points. 

[Drawing 61 It is drawing explaining an expansion trace of a fingertip. 

[Drawing 71 It is drawing explaining distributed processing. 

[Description of Notations] 

10 Desk 

12 Screen 

20 Pan Tilt Camera with Zoom Function 
30 Projector 

42 Surface Reflecting Mirror 
44 Surface Reflecting Mirror 
50 Infrared Camera 
60 Computer System 
62 Personal Computer 
64 Personal Computer 
66 SGI 02 and System 
68 Message Server 
70 Calibration Plate 
72 Miniature Lamp 
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